Papers with model initialization
Plasticity vs. Rigidity: The Impact of Low-Rank Adapters on Reasoning on a Micro-Budget (2026.eacl-srw)
Copied to clipboard
| Challenge: | Recent advances in mathematical reasoning typically rely on massive scale . yet, can strong reasoning capabilities be induced in small language models under extreme constraints? |
| Approach: | They train small language models with a single GPU for under 24 hours . they find that adapters unlock significant plasticity in standard instruction-tuned models . |
| Outcome: | The proposed model training on a single GPU (48GB) achieves 40% Pass@1 on AIME 24 (an 11.1% improvement over baseline) the model training results show that the adapter capacity and initialization are critical factors. |
An Effective Pronunciation Assessment Approach Leveraging Hierarchical Transformers and Pre-training Strategies (2024.acl-long)
Copied to clipboard
Bi-Cheng Yan, Jiun-Ting Li, Yi-Cheng Wang, Hsin Wei Wang, Tien-Hong Lo, Yung-Chang Hsu, Wei-Cheng Chao, Berlin Chen
| Challenge: | Existing attempts to quantify a second language learner’s pronunciation proficiency in a target language often sideline the hierarchy of linguistic units and relatedness among the pronunciation aspects. |
| Approach: | They propose a hierarchical automatic pronunciation assessment method that models the intrinsic structures of an utterance while considering the relatedness among the pronunciation aspects. |
| Outcome: | The proposed method can be used to quantify a second language learner’s pronunciation proficiency in a target language by providing fine-grained feedback with multiple pronunciation aspect scores at various linguistic levels. |
Discarding the Crutches: Adaptive Parameter-Efficient Expert Meta-Learning for Continual Semantic Parsing (2025.coling-main)
Copied to clipboard
| Challenge: | Continual Semantic Parsing (CSP) enables parsers to generate SQL from natural language questions in task streams, using minimal annotated data to handle dynamically evolving databases in real-world scenarios. |
| Approach: | They propose a Adaptive PET eXpert meta-learning approach that assists experts in adaptively warming up, ensuring better model initialization. |
| Outcome: | The proposed method outperforms existing methods on two benchmarks and achieves superior performance without data replay or ideal settings. |
Enhancing Automatic Readability Assessment with Pre-training and Soft Labels for Ordinal Regression (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models do not exploit ordinal nature of difficulty grades and make little effort for initialization to facilitate fine-tuning. |
| Approach: | They propose a readability assessment task that assigns a difficulty grade to a text . they use ordinal regression and pairwise relative text difficulty to train the model . |
| Outcome: | The proposed model outperforms competitive neural models and statistical classifiers on most datasets. |